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1 TITLE OF THE INVENTION 

2 STEREOSCOPIC IMAGE PROCESSING APPARATUS AND THE METHOD OF 

3 PROCESSING STEREOSCOPIC IMAGES 
4 

5 BACKGROUND OF THE INVENTION 

6 1 . Field of the invention 

7 The present invention relates to a stereoscopic image 

8 processing apparatus. More particularly, the present invention 

9 relates to a stereoscopic image processing apparatus in which 

10 so-called stereo matching is performed using a variable size of 

11 the small image region and to the method of such stereo matching. 

12 2 . Discussion of related arts 

13 In recent years, a stereoscopic image processing 

14 apparatus for calculating a distance to an object using a pair 

15 of picture images, has been put to practical use. The stereoscopic 

16 image processing apparatus calculates a positional deviation 

17 between a pair of images of an object, namely, a parallax. In 

18 calculating the parallax, a plurality of small image regions, 

19 that is, pixel blocks are provided in a lateral line on one 

20 reference image and the other comparison image, respectively and 

21 groups of pixel blocks positionally corresponding to each other 

22 are picked up from one frame of picture image (stereo matching) 

23 and a group of parallaxes is calculated for every frame of picture 

24 image. Distance data corresponding to each coordinate position 

25 of the picture image are calculated from the parallaxes. 
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1 In evaluating the correlation of brightness 

2 characteristic between reference and comparison pixel blocks, 

3 as disclosed in Japanese Patent Laid-open No . Toku-Kai 2002-267441 , 

4 the pixel block of a large area or size is advantageous from the 

5 point of the reliability of the stereo matching because of its 

6 increased number of pixels included in the block. 

7 However, in case of using the pixel block of a large 

8 size, there is a likelihood that the position of a target pixel 

9 block from which a parallax is calculated is deviated from the 

10 actual coordinate position corresponding to the parallax 

11 calculated with respect to the pixel block, because, when the 

12 correlation of brightness between the paired pixel blocks is 

13 evaluated, the position of the identified pixel blocks on the 

14 comparison image* - -is largely affected by a portion having a large 

15 brightness change. Accordingly, the pixel block having a large 

16 size produces erroneous correlations leading to a positional 

17 deviation of parallaxes. As a result, this positional deviation 

18 of parallaxes has a possibility of exacerbating the recognition 

19 accuracy of objects. In case of an object opposing to an axis 

20 of the camera, the recognition accuracy is not affected so much, 

21 however, in case of objects obliguely imaged like grounds, roads, 

22 the deviation of parallaxes has a serious effect leading to an 

23 erroneous recognition of grounds. 

24 Further, under adverse image conditions such as rains, 

25 nighttime and the like, since the image has weaker contrasts than 
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1 in fine weather, if the security of the stereo matching is a first 

2 priority , the pixel block having a large size is more advantageous . 

3 Under such situations as being able to obtain the security of 

4 the stereo matching, however, the small size pixel block is 

5 advantageous in consideration of the positional deviation of 

6 parallaxes. In prior arts, in processing the stereo matching, 

7 the size of the pixel block has been treated as unchanged in such 

8 a condition as neglecting either of these problems . 
9 

10 SUMMARY OF THE INVENTION 

11 It is an object of the present invention to provide 

12 a stereoscopic image processing apparatus capable of enhancing 

13 the reliability of the stereo matching and of calculating 

14 parallaxes with *high precision. 

15 A stereoscopic image processing apparatus for 

16 calculating a parallax between a pair of images comprise 

17 correlation evaluating means for evaluating a correlation of 

18 brightness between a reference pixel block provided in one of 

19 the pair of images and a comparison pixel block provided in the 

20 other of the pair of images and region size changing over means 

21 for changing over a size of the reference and comparison pixel 

22 blocks. According to a first aspect of the present invention, 

23 the size of the reference and comparison pixel blocks is changed 

24 over in accordance with an area where the reference pixel block 

25 is located. The size of those pixel blocks is changed over to 
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1 a large size when the comparison small region is located in a 

2 lower area of the image. 

3 According to a second aspect of the present invention, 

4 the size of the pixel blocks is changed over in accordance with 

5 imaging conditions such as rain, fog, snow, backlight, nighttime, 



6 snow on roads, stains or droplets on front windshield and the 

7 like. 
8 

9 BRIEF DESCRIPTION OF THE DRAWINGS 



10 Fig. 1 is a schematic block diagram showing a 

11 stereoscopic image processing apparatus according to a first 

12 embodiment of the present invention; 

13 Fig. 2 is an explanatory view showing a pixel block 

14 having a larger 'size than specified; 

15 Fig. 3 is an explanatory view of respective areas in 

16 which the size of the pixel block PBij is changed over; 

17 Fig. 4a is an explanatory view showing an example of 

18 a reference pixel block; 

19 Fig. 4b is an explanatory view showing an example of 

20 a comparison pixel block; 

21 Fig. 5a is an explanatory view showing an example of 

22 weighting factors established in a pixel block of 8 x 8 pixels; 

23 Fig. 5b is an explanatory view showing an example of 

24 weighting factors established in a pixel block of 4 x 4 pixels; 

25 Fig. 6 is a schematic block diagram showing a 
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1 stereoscopic image processing apparatus according to a second 

2 embodiment of the present invention; and 

3 Fig. 7 is a flowchart showing processes for changing 

4 over the size of a pixel block. 
5 

6 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

7 Referring now to Fig. 1, reference numeral 1 denotes 

8 a stereoscopic image processing apparatus functioning as a part 

9 of a vehicle surroundings monitoring apparatus. The stereoscopic 

10 image processing apparatus 1 recognizes circumstances in front 

11 of an own vehicle based on information given through stereoscopic 

12 image processes from a pair of images. 

13 A stereoscopic camera for imaging exterior sceneries 

14 is mounted in tfte vicinity of a room mirror of an own vehicle 

15 and is constituted by a pair of main and sub cameras 2, 3. The 

16 respective cameras 2, 3 incorporate an image sensor (CCD sensor, 

17 CMOS sensor or the like) . The main camera 2 takes reference (right ) 

18 images and the sub camera 3 takes comparison (left) image and 

19 these images are used for the stereoscopic image processing. 

20 Analogue images are outputted from the respective cameras 2, 3 

21 in amutually synchronized condition and are converted into digital 

22 images having a specified number of gradations (for example, 256 

23 gradations in gray scale) by A/D converters 4, 5. 

24 A pair of digitized image data are subjected to 

25 brightness corrections, geometrical conversions and the like in 
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1 an image correcting section 6. Generally, the installation of 

2 the paired cameras 2 , 3 in the vehicle is accompanied by errors 

3 to some extent and the left and right have deviations caused by 

4 these errors, respectively. In order to correct the deviations, 

5 the geometrical conversions such as rotations, parallel 

6 translations of images are performed using the Affine 

7 transformation. 

8 Through these image processes, reference images are 

9 obtained from the main camera 2 and comparison images are obtained 

10 from the sub camera 3. Thus obtained image data are constituted 

11 by a set of pixels having brightness ranging from 0 to 255. The 

12 image plane formed by the image data is expressed by the i-j 

13 coordinate system having an origin at the left bottom corner of 

14 the image. The i coordinate axis extends in a horizontal direction 

15 of the image plane and the j coordinate axis extends in a vertical 

16 direction. The stereoscopic image data for one frame are outputted 

17 to a stereoscopic image processing unit 7 which will be described 

18 hereinafter and stored in an image data memory 11. 

19 The stereoscopic image processing unit 7 is constituted 

20 by a stereo matching section 8, a parallax calculating section 

21 9 and a region control section 10 and calculates distance data 

22 of the picture image for one frame based on the reference image 

23 data and the comparison image data. The distance data are a group 

24 of parallaxes, that is, a set of parallaxes d calculated one by 

25 one for every reference pixel block PBij . The pixel block is an 



6 



1 image region having a specified size on the image plane formed 

2 by the reference image data and a calculation unit for one 

3 parallax. 

4 In calculating the parallax d with respect to a certain 

5 reference pixel block PBij on the reference image, a comparison 

6 pixel block (having the same size as that of the reference pixel 

7 block) of which brightness characteristic having a correlation 

8 with that of the reference pixel block PBij is identified in the 

9 comparison image. The distance to an object projected on the 

10 stereoscopic image is expressed as a parallax in the stereoscopic 

11 image, that is, a horizontal deviation between the reference 

12 image and the comparison image. Accordingly, in searching a 

13 comparison pixel block having a correlation in the comparison 

14 image, such comparison pixel block should be searched on the same 

15 horizontal line (epipolar line) as j coordinate. The stereo 

16 matching section 8 evaluates the brightness correlation between 

17 the reference pixel block and the comparison pixel block, while 

18 the comparison pixel block is shifted successively every one pixel 

19 on the epipolar line within a specified range of i coordinate 

20 established by the position of the reference pixel block PBij . 

21 Then, based on the result of the evaluation, the reference 

22 parallax calculating section 9 establishes the horizontal 

23 deviation of the comparison pixel block having a highest 

24 correlation from the reference pixel block to a parallax d of 

25 the pixel block PBij . 
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1 Thus, the stereo matching section 8 evaluates the 

2 correlation between the reference pixel block Pbij and the 

3 comparison pixel block using so-called "city block distance", 

4 one of correlation evaluation methods . The comparison pixel block 

5 having a correlation with the reference pixel block is located 

6 at a position where the city block distance CB becomes minimum. 

7 Since the parallax d is an essential parameter in 

8 transforming two-dimensional planes to three-dimensional spaces , 

9 the resolution of distance data, that is, the recognition ability 

10 of three dimensions rises with an increased number of the parallax 

11 data. Normally, the size of the pixel block PBij necessary for 

12 the stereo matching is around 4x4 pixels. For example, in case 

13 where the specified size of the pixel block PBij is established 

14 to 4x4 pixels, 5D--X 128 parallaxes are calculated from one frame 

15 of image. 

16 An increase of the number of pixels in the block, that 

17 is, the up-sizing of the pixel block PBij is preferable from the 

18 view point of increasing brightness information in order to 

19 ensure the stereo matching. As shown in Fig. 2, a pixel block 

20 PBij having 8x8 pixels for example can be obtained by enlarging 

21 the width of the pixel block having 4x4 pixels around the pixel 

22 block PBij by 4 pixels left and right, upward and downward, 

23 respectively. 

24 In up-sizing the pixel block PBij, however, there is 

25 a possibility that the coordinate position corresponding to the 
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1 parallax d of the pixel block. PBij differs from the coordinate 

2 position corresponding to the calculated parallax Z of the enlarged 

3 pixel block. Basically, since the three-dimensional recognition 

4 using the stereo matching is performed on the basis of the parallax 

5 d and the coordinate position corresponding to the parallax d 

6 of the pixel block PBij , the positional deviation on coordinates 

7 of the calculated parallax d may cause loose recognitions of 

8 vertical and horizontal positions. Particularly, objects not 

9 opposite to the image plane, for example, roads, lane markers 

10 and the like, are apt to be recognized loosely. 

11 One of the features of the embodiment is that the region 

12 control section 10 changes the size of the reference pixel block 

13 PBij and the comparison pixel block in evaluating the correlation 

14 between the reference pixel block and the comparison pixel block. 

15 That is, the region control section 10 can vary the size of the 

16 referencepixel block PBij according to the area where the reference 

17 pixel block PBij is located. In this embodiment, the size of the 

18 reference pixel block PBij is varied according to whether objects 

19 are projected in an opposed manner to the image plane or objects 

20 are projected obliquely with respect to the image plane. 

21 Referring to Fig. 3, in this embodiment, a boundary 

22 line BL extending horizontally is provided on the image plane 

23 and the size of the pixel block PBij is controlled differently 

24 at the respective areas divided by the boundary line BL . 

25 Specifically, the size of the pixel block PBij is established 
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1 to 8 x 8 pixels at the area above the boundary line BL and the 

2 size of the pixel block PBij is established to 4 x 4 pixels at 

3 the area below the boundary line BL. The position of the boundary 

4 line BL is determined by a boundary determination section 15 which 

5 will be described hereinafter. The area below the boundary line 

6 BL is an area on which grounds, roads and the like are projected 

7 and the area above the boundary line BL is an area on which solid 

8 objects existing on the ground are projected. 

9 Fig. 4a shows an example of a reference pixel block 

10 PBij having 8 x 8 pixels and Fig . 4b shows an example of a comparison 

11 pixel block PBij having 8x8 pixels. The city block distance 

12 CB is expressed as follows: 

13 [Formula 1] 

14 CB = E(|A33 -B331 + + |A66 - B66 | ) 

15 + KS[(|A11 -Bill + +IA88 - B88 | ) 

16 -(|A33 -B33| + + |A66 - B66 1 ) ] 

17 where A11---A88 are brightness values of respective pixels pij 

18 in the reference pixel block PBij and Bll- • -B88 are brightness 

19 values of respective pixels pij in the comparison pixel block 

20 PBij . 

21 In the formula 1 , the first term is the sum of the absolute 

22 values of differences of brightness values Aij , Bij positionally 

23 corresponding in a central region (4x4 pixels) and the second 

24 term is the sum of the absolute values of differences of brightness 

25 values Aij , Bij positionally corresponding in a surrounding 
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1 region (obtained by subtracting central region from 8 x 8 pixels ) . 

2 The city block distance CB is calculated by adding the first term 

3 and the second term having a coefficient K. The coefficient K 

4 is given from the region control section 10 to the stereo matching 

5 section 8 and changes over from 1 to 0 and vise versa according 

6 to the position of the pixel block PBij and the position of the 

7 boundary line BL. Specif ically , when the pixel block PBij is located 

8 at the area above the boundary line BL, the region control section 

9 10 outputs an instruction that the coefficient K should be 1 . 0 

10 and when the pixel block PBij is located at the area below the 

11 boundary line BL, the region control section 10 outputs an 

12 instruction that K should be 0. 

13 In case where K = 1 is given, the stereo matching 

14 section 8 evaluates the correlation with the pixel block PBij 

15 having 8x8 pixels (hereinafter referred to as the 8 x8 pixel 

16 block) . Further, in case where K=0 is given, the stereo matching 

17 section 8 evaluates the correlation with the pixel block PBij 

18 having 4x4 pixels (hereinafter referred to as the 4x4 pixel 

19 block) . Thus, the region control section 10 acts as giving two 

20 sizes of the pixel block PBij ,4x4 pixels for a small size and 

21 8x8 pixels for a large size. 

22 In this embodiment , the number of calculated parallaxes 

23 d is egual to the number of the pixel blocks PBij having 4x4 

24 pixels. Accordingly, in case where the 4x4 pixel blocks PBij 

25 are used according to the region control, after the evaluation 
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1 of correlation is finished for a given pixel block PBij , as shown 

2 in Fig. 2, the next object of evaluation is an adjacent pixel 

3 block Pbi+lj . On the other hand, in case where the 8x8 pixel 

4 blocks PBij are used according to the region control, after the 

5 evaluation of correlation is finished for a given pixel block 

6 PBij, as shown in Fig. 2, the next object of evaluation shifts 

7 to a 8 x8 pixel block PBi+lj horizontally overlapped by two pixels 

8 with the pixel block PBij . That is, the center of the pixel block 

9 PBij is away from the center of the pixel block Pbi+lj by four 

10 pixels. Similarly, the next object of evaluation is a pixel block 

11 PBij+1, this pixel block PBij+1 is a pixel block having 8x8 

12 pixels vertically overlapped by two pixels with the pixel block 

13 PBij . 

14 In cas.e where the 8x8 pixel block PBij is used, 

15 sometimes the surrounding region of the pixel block PBij goes 

16 out of the effective image region at the edge of the four sides 

17 of the image. In this case, the correlation is evaluated with 

18 the small size, 4x4 pixel block PBij . The blank space is provided 

19 along the circumferential edge of the region above the boundary 

20 line BL for that purpose. 

21 Thus, the stereo matching section 8 calculates the city 

22 block distance CB for every pixel block PBij using the pixel block 

23 PBij having 4 x 4 or 8 x 8 pixels, by means of horizontally shifting 

24 the comparison pixel block successively by one pixel. Further, 

25 the parallax calculating section 9 selects the comparison pixel 
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1 block having the smallest city block distance and calculates the 

2 horizontal deviation between the reference pixel block PBij and 

3 the comparison pixel block as the parallax d. The stereoscopic 

4 image processing unit 7 calculates the parallaxes d for the entire 

5 image of one frame size, while changing the size of the pixel 

6 block PBij . Thus obtained distance data D are stored in a distance 

7 memory 12 . 

8 Reference numeral 13 denotes a micro-computer 

9 constituted by CPU, ROM, RAM, Input/Output interface and the like . 

10 In order to understand functionally, the micro-computer 13 is 

11 constituted of a recognition section 14 and a boundary 

12 determination section 15. 

13 Based on respective information stored in the image 

14 data memory 11 ahd the distance data memory 12, the recognition 

15 section 14 recognizes roads and the like ahead of the own vehicle 

16 (road profiles recognition) or recognizes solid objects like 

17 vehicles in front of the own vehicle (solid objects recognition) . 

18 The road profiles recognition is to express 

19 three-dimensional road profiles by the function concerning left 

20 and right lane markers, no passing markers and the like and to 

21 establish respective parameters to such values as agreeing with 

22 actual roadprofiles ( straight road , curved road, or ups and downs) . 

23 For example, the recognition section 14 read reference image data 

24 from the image data memory 11 and identifies an object caused 

25 by markers from marker edges, that is, horizontal brightness edges 
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1 (portion having a large variation of brightness between images 

2 adjacent to each other) . 

3 Substituting the coordinates (i, j) and the parallax 

4 d into a known coordinate conversion formula for every identified 

5 marker edge, coordinates (x,y, z) in the actual space established 

6 on the own vehicle is calculated. Thus established coordinate 

7 system has an origin on the road surface directly underneath the 

8 center of two cameras (stereoscopic camera) , x axis extending 

9 in a widthwise direction of the own vehicle, y axis extending 

10 in a vertical direction of the vehicle and z axis extending in 

11 a lengthwise direction of the vehicle. Marker models are prepared 

12 based on the coordinates (x, y, z) of the respective marker edges 

13 in the actual space. That is, first, an approximation line is 

14 obtained for every specified interval with respect to the 

15 respective left and right marker edges extending forwardly. A 

16 marker model is expressed like a folded line by connecting thus 

17 obtained approximation lines with each other . Further, the marker 

18 model is constituted by a curve function (X = f (Z) ) representing 

19 a curvature of the curve and a grade function (Y = f(Z)) representing 

20 a grade or an up and down. Thus, the three dimensional conditions 

21 of a road in the real space can be expressed by the left and right 

22 marker models . 

23 Further, the recognition section 14 identifies the data 

24 above road surfaces as candidates of solid objects based on the 

25 detected road profiles (marker models) . The candidates having 
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1 similar distances in terms of z and x directions are treated as 

2 one group of solid objects. That is, the recognition section 14 

3 identifies the group of solid objects located on the same lane 

4 as a preceding vehicle and identifies the group of solid objects 

5 located outside of the lane is identified as an oncoming vehicle, 

6 an obstacle or a side wall. Further, the recognition section 14 

7 has a function of warning a driver by operating a warning device 

8 in case where it is judged that a warning is needed based on the 

9 result of the recognition. Also, the recognition section 14 has 

10 a function of controlling a vehicle by shifting down an automatic 

11 transmission, reducing engine power, or applying brakes based 

12 on the result of the recognition. 

13 The boundary determination section 15 determines the 

14 boundary line BEv.for dividing the image plane into a plurality 

15 of areas. In this embodiment, the image plane is divided by a 

16 boundary line BL extending horizontally into two areas, an area 

17 on which road surfaces are projected and an area in which a solid 

18 object exists on the road surfaces. The position of the boundary 

19 line BL may be determined according to the statistics method, 

20 however, since the vehicle pitches and the road has ups and downs, 

21 it is preferable that the position of the boundary line BL is 

22 variable. According to the embodiment of the present invention, 

23 the boundary determination section 15 controls the position of 

24 the boundary line BL . 

25 For that purpose, the boundary determination section 
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1 15 must know the present position of the ground surface. 

2 Specifically, first, the boundary determination section 15 

3 calculates a vanishing point from a point of intersection of two 

4 left and right lane markers on the image plane based on the road 

5 profile recognized by the recognition section 14, using the 

6 parallelism of the lane markers. The boundary position between 

7 the ground and solid objects is determined to be located at a 

8 position downwardly offset by a few pixels. The boundary line 

9 BL is provided at the position. At this moment, the boundary 

10 determination section 15 outputs parameters for determining the 

11 boundary line BL to the region control section 10. 

12 As understood from the description above, according 

13 to the method of processing stereoscopic images described in the 

14 first embodiment^ the size of the pixel block PBij can be changed 

15 over between the small size and the large size in accordance with 

16 the area in the image plane. Hence, the correlation is evaluated 

17 with the 8x8 pixel block PBij in the area on which solid objects 

18 are projected and the correlation is evaluated with the 4x4 

19 pixel block PBij in the area on which road surfaces are projected. 

20 As a result, the large size pixel block PBij can ensure the stereo 

21 matching. On the other hand, the small size pixel block PBij 

22 prevents deviations of parallaxes caused by the large size of 

23 the pixel block PBij. As a result, the recognition accuracy of 

24 the road surface, particularly in a horizontal and vertical 

25 direction can be ensured. 
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1 In this embodiment, the size of the pixel block PBij 

2 is changed over between two sizes, 4x4 pixels and 8x8 pixels, 

3 however, the size of the pixel block PBij is not limited to these 

4 sizes. Appropriate number of pixels can be used as desired. 

5 Further, in this embodiment, the image plane is divided into two 

6 regions by the boundary line BL, however, the image plane may 

7 be divided into three or more areas by a plurality of boundary 

8 lines . In this case , the size of the pixel block PBij may be changed 

9 over according to the area. 

10 In the aforesaid embodiment, the size of the pixel block 

11 PBij is changed over by changing over the value of K in the formula 

12 1 . On the other hand, the city block distance CB can be defined 

13 as the sum of weighted absolute value of the difference between 

14 two brightness Values Aij , Bij corresponding to each other for 

15 the entire pixel blocks PBij . 

16 CB = 2(wll|All -Bill + + w88|A88 - B88|) (2) 

17 Fig . 5a shows an example of the pixel block PBij obtaining 

18 the size of 8x8 pixels by applying the weight factor wij = 1 

19 to each pixel and Fig. 5b is an example of the pixel block PBij 

20 obtaining the size of 4 x 4 pixels by applying the weight factor 

21 wij = 1 to each pixel of the central region and wij = 0 to each 

22 pixel of the surrounding region. The change of the weight factor 

23 wij is instructed from the region control section 10 in the same 

24 manner as the change of the aforesaid coefficient K. 

25 In case where the correlation of the pixel block PBij 
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1 is evaluated with the large size (8x8 pixels) of the pixel block 

2 PBij , as the matching goes apart from the center of the pixel 

3 block, the reliability of the comparison image data declines. 

4 To prevent this, the weight factors may be established to 1.0 

5 in the central region and at the same time the weight factors 

6 may be gradually decreased in the surrounding region. 

7 Fig. 6 is a schematic block diagram showing a 

8 stereoscopic image processing apparatus la according to a second 

9 embodiment of the present invention. In the second embodiment, 

10 the components of the stereoscopic image processing apparatus 

11 la which are identical to the first embodiment are denoted by 

12 identical reference numbers and are not described in detail . 

13 The difference of the second embodiment from the first 

14 embodiment is to change over the size of the pixel block according 

15 to imaging circumstances. Specifically, the region control 

16 section 10 changes over the size of the pixel block PBij from 

17 the small size (4x4 pixels) to the large size (8x8 pixels) 

18 in surrounding conditions such as rain , nighttime, smog, backlight 

19 snowy roads and the like, regardless of the area of the image. 

20 Because , since images taken in these conditions have small contrast 

21 using the large size pixel block PBij is advantageous in securing 

22 the stereo matching. 

23 Referring to Fig. 7, the routine of the flowchart is 

24 energized at a specified interval and executed by the region control 

25 section 10. First, at a step 1, it is judged whether or not the 
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1 weather is rain by a wiper switch (not shown) turned on or off. 

2 Otherwise, the judgment of rainy condition can be made based on 

3 the number of the data isolated from the grouped solid object 

4 data. Further, otherwise, the rainy condition can be judged from 

5 the number of mismatches which occurs when the brightness of the 

6 reference pixel block PBij coincides with the brightness of a 

7 positionally unrelated comparison pixel block. That is, in the 

8 rainy condition, droplets on the front windshield of the vehicle 

9 or raindrops increase isolated data or mismatches. Further, if 

10 it is detected that the wiper switch is turned on, more accurate 

11 judgment of the rainy condition can be expected. 

12 In case where the judgment is affirmative, namely, in 

13 case where images are taken in the rainy condition, the program 

14 goes to a step 7 and in case where the judgment is negative, namely, 

15 in case of no rain, the program goes to a step 2. 

16 At the step 2, it is judged whether or not images are 

17 taken in nighttime . The nighttime condition is j udged from whether 

18 a head light switch is turned on or off. Otherwise, the nighttime 

19 condition can be j udged from the guantity of exposure of the cameras 

20 2,3. The quantity of exposure is calculated by an exposure control 

21 section (not shown) . The exposure control section controls 

22 parameters (for example, shutter speeds, apertures of lenses and 

23 amplification gains) for adjusting the guantity of exposure so 

24 as to obtain an optimum brightness at the next frame of image 

25 based on the present brightness value. It is possible to judge 
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1 the nighttime condition by using these parameters as a judgment 

2 criteria. Further, it is possible to j udge the nighttime condition 

3 more accurately by using the result of the judgment together with 

4 the ON/ OFF condition of the headlight switch. 

5 If the judgment of the nighttime condition is 

6 affirmative , that is , in case where images are taken in the nighttime , 

7 the program goes to the step 7 and if it is negative, that is, 

8 in case where it is not the nighttime condition, the program goes 

9 to a step 3 . 

10 Judgments of smog, backlight and snowy road are made 

11 at steps 3 , 4 and 5, respectively. The judgment of smog is performed 

12 based on the number of the distance data (parallax d) within a 

13 specified region and the number of the brightness edges in the 

1 4 image . The j udgme*nt of backlight is performed based on the quantity 

15 of exposure of the cameras 2, 3, the average brightness values 

16 at the upper part of the image and the average brightness values 

17 at the lower part of the image. Further, the judgment of snowy 

18 road is performed based on the average brightness or the number 

19 of brightness edges within a specified road surface area. 

20 In case where all judgments from the step 1 to 5 are 

21 negative, since it is judged that the imaging condition is not 

22 so bad as needing the large size of the pixel block PBij , the 

23 size of the pixel block PBij is established to 4 x 4 pixels. At 

24 this moment, the region control section 10 changes over the size 

25 of the pixel block from 8x8 pixels to 4 x 4 pixels. In case 
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1 where the size of the pixel block is already 4x4 pixels, that 

2 size is maintained. 

3 On the other hand, in case where either of the steps 

4 1 to 5 is affirmative, it is judged that the imaging condition 

5 is so bad as needing the large size of the pixel block for securing 
• 6 the stereo matching and the size of the pixel block PBij is 

7 established to 8 x 8 pixels. At this moment, the region control 

8 section 10 changes over the size of the pixel block from 4x4 

9 pixels to 8 x 8 pixels. In case where the size of the pixel block 

10 is already 8x8 pixels, that size is maintained. 

11 Thus, after the size of the pixel block PBij is 

12 determined, the stereoscopic image processing unit 7 calculates 

13 the city block distance CB for every comparison pixel block PBij 

14 over the entire*-- specif ied searching area while shifting the 

15 comparison pixel block PBij by one pixel horizontally/vertically 

16 and determines the comparison pixel block having the correlation 

17 in brightness between the reference and comparison pixel blocks. 

18 The stereoscopic image processing unit 7 calculates the parallaxes 

19 d successively with respect to the image of one frame. 

20 As described above , according to the second embodiment , 

21 since the optimum size of the pixel block PBij to be evaluated 

22 can be selected in accordance with the imaging conditions such 

23 as rain, smog, nighttime and the like, more accurate 

24 three-dimensional recognitions can be accomplished for the entire 
2 5 image of one frame. 
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1 In the embodiments described above, rain, nighttime, 

2 smog, snow on road are exemplified as bad imaging conditions, 

3 however , the present invention is not limited to these . Sandstorm, 

4 darkness in tunnels, stain on windshields, stain on lenses of 

5 the cameras 2 , 3 etc . may be included in the bad imaging conditions . 

6 The entire contents of Japanese Patent Application No. 

7 Tokugan 2002-282644 filed September 27, 2002, is incorporated 

8 herein by reference. 

9 While the present invention has been disclosed in terms 

10 of the preferred embodiments in order to facilitate better 

11 understanding of the invention, it should be appreciated that 

12 the invention can be embodied in various ways without departing 

13 from the principle of the invention. Therefore, the invention 

14 should be understood to include all possible embodiments which 

15 can be embodied without departing from the principle of the 

16 invention set out in the appended claims. 
17 



22 



